1. Field of the Invention
Embodiments of the present invention relate to multi-spectral imaging systems such as still cameras, video cameras, scanners and microscopes and more specifically to imaging systems that use fewer sensor elements than previous techniques for comparable image quality.
2. Background Information
Images herein can be considered signals whose amplitude may represent some optical property such as intensity, color and polarization which may vary spatially but not significantly temporally during the relevant measurement period. In color imaging, light intensity typically is detected by photosensitive sensor elements or photosites. An image sensor is composed of a two dimensional regular tiling of these individual sensor elements. Color imaging systems need to sample the image in at least three basic colors to synthesize a color image. We use the term “basic colors” to refer to primary colors, secondary colors or any suitably selected set of colors. We exclude color difference signals, many of which are used in popular color spaces, from the definition of basic colors. Furthermore, all references to red, green and blue should be construed to apply to any set of basic colors.
We only consider systems that sense a “substantially local” transform in the spatial domain of the image signal, which is a transform whose value at a point depends only on the signal in its close neighborhood.
Color sensing may be achieved by a variety of means such as, for example, (a) splitting the image into three identical copies, separately filtering each into the basic colors, and sensing each of them using separate image sensors, or (b) using a rotating filter disk to transmit images filtered in each of the basic colors onto the same image sensor.
However, a very popular design for capturing color images is to use a single sensor overlaid with a color filter array (“CFA”). This includes the straightforward design wherein the value of each output pixel is determined by three sensing elements, one for each basic color, usually arranged in horizontal, vertical or diagonal stripes. This design yields red, green and blue images of equal resolution, or equivalently luminance and chrominance signals of equal bandwidth. ‘Luminance” is defined as a weighted sum of basic color signals where all the weights are positive while “chrominance”, is defined as a weighted sum of basic color signals where at least one weight is negative. The color stripe design is still used in high end cameras such as the Panavision Genesis Digital Camera, http://www.panavision.com/publish/2007/11/09/Genesis.pdf, page 2, 2007. Newer CFA designs by Bayer (see FIG. 4 and B. E. Bayer, “Color imaging array”, Jul. 20, 1976. U.S. Pat. No. 3,971,065) and others (see K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and L. Condat, “A New Class of Color. Filter Arrays with Optimal Sensing Properties”) make different trade-offs between luminance and chrominance bandwidths as well as the crosstalk between them.
Most of the early CFAs as well as their associated image reconstruction algorithms were designs based on intuition and experimentation but lacking rigorous mathematical foundation. In the paper “Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain”, Proc. IS&T/SID 10th Color Imaging Conf, pages 331-336, 2002, D. Alleysson, S. Susstrunk, and J. Herault analyzed electromagnetic filtering performed by CFAs as amplitude modulation of the color signals in the spatial domain (as used herein the terms “demosaic” and “demosaick” are to be construed as input image reconstruction procedures and the terms “demosaicer” and “demosaicker” as input image reconstruction algorithms). This led to frequency domain image reconstruction techniques that viewed the problem as that of demultiplexing the luminance and chrominance signals via demodulation and filtering. See E. Dubois, “Frequency-domain methods for demosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 200 and N. Lian, L. Chang, and Y. P. Tan, “Improved color filter array demosaicking by accurate luminance estimation” in IEEE International Conference on Image Processing, 2005, ICIP 2005, volume 1, 2005.
The complementary problem of designing CFAs with good frequency domain properties was attacked by D. Alleysson, S. Susstrunk, and J. Herault, “Linear demosaicing inspired by the human visual system”, IEEE Transactions on Image Processing, 14(4):439-449, 2005 wherein the doubling of the number of blue photosites in the Bayer CFA at the expense of Green photosites was suggested. This was followed by techniques to design CFAs directly in the frequency domain by K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and optimized by L. Condat, “A New Class of Color Filter Arrays with Optimal Sensing Properties”. These techniques fix the pattern of each basic color to consist of a small set of spatial “carriers”—two dimensional sinusoids with appropriate frequencies, phases and amplitudes—and sum over the three basic colors to arrive at the final pattern. This pattern is then overlaid on the sensor. When an image formed by the camera's lens is filtered by the CFA, it is modulated by each of the carrier frequencies. The overlap of the modulation products of the 3 primaries induces a color transform and leads to a multiplex of luminance and chrominance signals modulated at different frequencies. As long as there is limited cross-talk between the luminance and chrominance signals, and the color transform is invertible the original color image can be recovered.
An important consideration in the choice of sensor color space so far has been the high frequency content of chrominance signals. Well chosen color transforms result in chrominance signals with low high frequency content. This allows the chrominance signals to be placed close to each other and to the luminance signal in the Fourier domain without significant cross-talk. See Y. Hel-Or, “The canonical correlations of color images and their use for demosaicing” and K. Hirakawa and P. J. Wolfe, “Spatio-spectral color filter array design for enhanced image fidelity” in Proc. of IEEE ICIP, pages II: 81-84, 2007 and L. Condat, “A New Class of Color Filter Arrays with Optimal Sensing Properties”.
An important factor influencing the close packing of color component (luminance, chrominance) signals is the geometry of their spectra. Square and rectangular sampling lattices admit higher resolution along the diagonal directions than along horizontal or vertical directions. Optical systems, on the other hand, generate roughly equal resolution in all directions thereby yielding images with nearly circular spectral support. When plotted, the 2D Fourier transform of a CFA is a rectangle while that of a color component of an optical image is a circle. This leads to the problem of efficiently packing circles into rectangles. FIG. 3 shows an exemplary monochrome image 310 and it's spectral image 320. The spectral image is obtained by taking the logarithm of the absolute value of the Fourier transform of the image.
An aggressive class of techniques for close packing of color component spectra employs adaptive directional techniques during image reconstruction. These techniques assume the color component spectra of small image patches to be sparse in at least one direction. They design their CFA to generate more than one copy of chrominance spectra (see B. E. Bayer, “Color imaging array”, Jul. 20, 1976, U.S. Pat. No. 3,971,065), implicitly or explicitly identify the cleanest copy during the image reconstruction step and use directional filtering to demultiplex them (see E. Dubois, “Frequency-domain methods for demosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 2005 and K. Hirakawa and T W Parks, “Adaptive homogeneity-directed demosaicing algorithm”, IEEE Transactions on Image Processing, 14(3):360-369, 2005. Also see Ron Kimmel, “Demosaicing: Image reconstruction from color ccd samples”, IEEE Trans. Image Processing, 8:1221-1228, 1999 and E. Chang, S. Cheung, and D. Y. Pan, “Color filter array recovery using a threshold-based variable number of gradients”, in Proceedings of SPIE, volume 3650, page 36, 1999). The benefits of adaptive directional image reconstruction come at a heavy cost, though, since sensing edge directions from noisy sub-sampled images is a hard problem and the non-linear nature of sensing edges making makes noise reduction a non-separable step from image reconstruction.
Universal demosaickers have also been devised that can reconstruct images sampled by any CFA. This has allowed for easy experimentation with novel CFA designs, including patterns with basic colors arranged in a random pattern (as used herein the term “random” is to be construed as including “random” and “pseudo random”).
CFA Design in the Fourier Domain
Consider a photosite located at n=[n1 n2] that filters incident light x(n)=[xr(n) xg(n) xb(n)]T through color filter array c(n)=[cr(n) cg(n) cb(n)] and measures the resulting, scalar signal y(n), wherey(n)=c(n)·x(n)  (1)
Consider a set of real carrier sinusoids s(k)(n), 1≦k≦m of unit amplitude, frequencies ω(k)=[ω1(k),ω2(k)] and phases
     ⁢                    ϕ                  (          k          )                    ∈              {                  0          ,                      π            2                          }              ,  given by
                                          s                          (              k              )                                ⁡                      (            n            )                          =                                            ⅇ                              j                ⁡                                  (                                                                                    ω                                                  (                          k                          )                                                                    ·                      n                                        +                                          ϕ                                              (                        k                        )                                                                              )                                                      +                          ⅇ                              -                                  j                  ⁡                                      (                                                                                            ω                                                      (                            k                            )                                                                          ·                        n                                            +                                              ϕ                                                  (                          k                          )                                                                                      )                                                                                2                                    (        2        )            Each color of the CFA, ci(n), iε{r, g, b}, is the superposition of these carriers scaled by an appropriate real amplitude αi(k),
                                          c            i                    ⁡                      (            n            )                          =                              ∑                          k              =              1                        m                    ⁢                                    a              i                              (                k                )                                      ⁢                                          s                                  (                  k                  )                                            ⁡                              (                n                )                                                                        (        3        )            
The choice of carrier frequencies is a CFA design decision except for the zero frequency or DC component, whose presence is essential for all physically realizable CFAs. For this reason we set ω(1)=[0 0]. It follows that αi(1)>0, iε{r, g, b}.
Once the sensor is exposed to image x(n) and its mosaiced output y(n) is captured, a signal processing step is needed to reconstruct x(n). Assuming the carrier frequencies ω(k), 1≦k≦m are sufficiently separated so that sidebands centered about them do not overlap, each modulated signal can be recovered by multiplication with its respective carrier followed by convolution with a low pass filter h(k). Formally,u(k)(n)=(h(k)*(s(k)·y))(n)  (4)
Each u(k)(n), 0≦k≦m can be viewed as a color component. Motivated by the fact that αi(1)>0, iε{r, g, b}, we loosely refer to u(1)(n) as the luminance signal, and u(k)(n), k>1 as the chrominance signals.
It is important to note that if the carrier frequencies ω(k), 1≦k≦m are insufficiently separated so that all sidebands centered about them overlap, existing techniques do not prescribe any means of recovering them without crosstalk.
Since u(n)=[u(1)(n) u(2)(n) . . . u(m)(n)]T is generated by the modulation of the incident image x(n), it can be written asu(n)=A·x(n)  (5)where
  A  =            [                                                  a              r                              (                1                )                                                                        a              g                              (                1                )                                                                        a              b                              (                1                )                                                                                        a              r                              (                2                )                                                                        a              g                              (                2                )                                                                        a              b                              (                2                )                                                                          ⋮                                ⋮                                ⋮                                                              a              r                              (                m                )                                                                        a              g                              (                m                )                                                                        a              b                              (                m                )                                                        ]        .  A can be interpreted as the color transform matrix, and provided its rank is 3 or greater, x can be recovered byx(n)=A−1·u(n)  (6)Here A−1, the generalized inverse of A, can be interpreted as the inverse color transform.
From the above discussion it is clear that the three decision variables for a CFA design are the carrier frequencies ω(k), 1≦k≦m, phases φ(k), 1≦k≦m and amplitudes given by the matrix A.
Bayer CFA in the Fourier Domain
For a Fourier domain analysis of the popular Bayer CFA see E. Dubois, “Frequency-domain methods for de-mosaicking of bayer-sampled color images”, IEEE Signal Processing Letters, 12(12):847-850, 2005. FIG. 4 shows the Bayer CFA 410. FIG. 5 illustrates how color information with its circular support is packed into the sensor's rectangular support. This can be most easily understood in terms of an alternative color space:
                              [                                                    L                                                                                      C                  ⁢                                                                          ⁢                  1                                                                                                      C                  ⁢                                                                          ⁢                  2                                                              ]                =                                            1              4                        ⁡                          [                                                                    1                                                        2                                                        1                                                                                                              -                      1                                                                            2                                                                              -                      1                                                                                                                                  -                      1                                                                            0                                                        1                                                              ]                                ⁡                      [                                                            R                                                                              G                                                                              B                                                      ]                                              (        7        )            
In FIG. 5, the central circle represents Luminance (L). The four quarter circles at the vertices make up Chrominance1 (C1). The two semi-circles at the left and right edges make up the first copy of Chrominance2 (C2a). The two semi-circles at the top and bottom edges make up the second copy of Chrominance2 (C2b).
It's apparent from this figure that there is empty space between circles that goes unused. Such inefficiencies are unavoidable by existing CFAs that attempt to maintain at least one copy of luminance or chrominance spectra free from crosstalk. This invention redresses such inefficiency, among other things.
Universal Demosaickers
A few so called universal image reconstruction methods exist which work with arbitrary Red, Green and Blue CFA patterns. These include one described in Condat, “Random patterns for color filter arrays with good spectral properties”, (Research Report of the IBB, Helmholtz Zentrum Munchen, no. 08-25, September 2008, Munich, Germany), IBR, hereby incorporated by reference in its entirety, which uses variation minimization to infer the original image.
Lukac et al., “Universal demosaicing for imaging pipelines with a RGB color filter array” (Pattern Recognition, vol. 38, pp. 2208-2212, 2005) IBR, hereby incorporated by reference in its entirety, presents a universal demosaicker which makes assumptions about the local constancy of color ratios.
Neither of these universal demosaickers use spectral band-limitedness constraints in a systematic way.